1. Technical Field of the Invention
The present invention most generally relates to computer animation, and more specifically to an animation system that blends motion capture data into a customizable work incorporating motion capture data that produces animation that is much closer to real life movements.
2. Background of the Invention
Animation generally refers to the combination of graphics with the dimension of time. When single images are sequenced together, it gives the appearance of movement. Prior to computers, animators drew hundreds of images to represent a single cartoon movement. Computers now automate the drawing of the images to some extent and aid in manipulating the images to portray certain movements.
Computer animation is well known in the art, and the animator uses the processing power of the computer to create images and motion sequences that represent the desired animation. While it is relatively simple to create jagged and unrealistic movements, generating interactive sequences of animation that follow natural human/animal motion is difficult to create.
To produce quality computer animation, it is necessary to study the motions of the object being represented and form an animation program on the basis of the analysis. Animation that perform the motions of a human require observation by the animator. On the basis of the observations, a program for producing the same motions by the computer model is formed. But, observations are subjective and based on the understanding of the animator. Therefore, the animator""s representation always incorporate some deviation from reality.
An animator applies kinematics to the motions, which describes the motions only in terms of positions, rotation, scaling, velocities and accelerations, neglecting the forces and torques responsible. Kinematics analysis only generates a line picture representing parts constituting a human or animal body, and a three-dimensional model of the human or animal body cannot be displayed realistically on a screen.
Keyframing has been used to produce animation that involves identifying the position of body parts and interpolating between the positions of the body parts in successive key frames to generate the intermediate positions of the body parts over the time interval of the animation. Forward kinematic models have generally been discussed in relation to the interpolation techniques. This concept of keyframing builds motion sequences frame by frame and is fairly costly and time consuming. In addition, this method requires expertise to generate quality realistic motions.
Another method of computer animation, motion capture, users sensors attached to a subject which are used to capture data representing the positions of the joints/body parts over time during the performance of a desired motion. The result is a recorded motion sequence for a particular motion. This recorded motion drives a forward kinematic model to produce a desired animation. There are several disadvantages in the prior art associated with motion capture, namely the recorded motion data is sometimes limited by the motions of the recorded subject.
Inverse kinematics (IK) for computer animation has also been used to a limited extent. Inverse kinematics processes a desired position in three-dimensional space and is used to calculate body joint angles. In this approach the particular body part is moved to the desired position resulting from the computer processing. Inverse kinematics often is sometimes used with keyframing to in manipulating the subject into the desired positions with animated character movements.
The IK methodology is based upon complex matrix manipulations that require significant amounts of processing to determine the joint angles associated with limb movements. The amount of processing increases as a cubic of the number of joints involved, translating into lengthy processing times when numerous joints are involved. A human being has more than 200 degrees-of-freedom, which makes it practically impossible to use matrix-based inverse kinematic methods to interactively animate any realistic human or animal-like character in real time. And, matrix-based IK methods may not even work on certain joint configurations known as singular configurations. Furthermore, the prior art notes that with multiple limb and joint movements, the end result of the computations will not appear natural.
In addition, prior IK methods must converge to a solution before the results of the computation can be used for the animation. Partially computed results causes unstable, jerky and oscillatory motion of the limbs, with large positioning errors.
Animation based on keyframing and motion capture is also not highly successful with interactive applications, such as video games and virtual reality simulators. The virtual world of interactive video gives a user total freedom within a three dimensional (3D) space. But, the animator typically employs existing animation and works within the realm most familiar, producing animated characters that all move in like manner.
Besides resulting in poor quality and non-original movement sequences, the existing technology is time consuming and laborious. The frames are created for each movement, and there must be numerous frames with slight movements or the resulting animation appears to move in a jerky fashion. In order to reduce the workload, animators usually limit the number of moving parts. However life-like movement is usually the result of multiple joint and limb movement and using a lesser degree of movement results in an unnatural animation.
An alternative to key-framing and motion capture techniques to create animation is behavioral animation. Behavioral animation refers to movements generated in response to commands and interaction with the environment. This method of animation relies more heavily on computer processing to determine the movement sequence to accomplish the command. High-level goal-based behaviors are typically made possible by using a combination of forward kinematics and inverse kinematics. One advantage of behavioral animation is that the movements can be created on-line and modified in real-time.
Dynamics provides another method of motion analysis. Dynamics analyzes motions of objects based on the relation between movements and forces, as compared to kinematics, which provides motion analysis in terms of positions, velocities and accelerations. Dynamics allows more complex behavioral or animated results. But, computer analysis of motions utilizing dynamics requires data on parameters such as the moments of inertia, the centers of gravity, joint friction and muscle/ligament elasticity of the moving body being represented by the animation. Dynamics motion analysis also requires complex dynamics equations of multiple and simultaneous differential equations.
Solving a problem based on dynamics involves differential equations that describe the relationship between mass and force and torque. There are a number of equations used for describing dynamics, but the most familiar formulations is Lagrange""s equation:
∂L/∂qixe2x88x92d/dt∂L/∂qi+xcex∂f/∂qi+Qi=0
where L is the Lagrangian given by kinetic energy (T) minus potential energy (P), qi is the ith coordinate, f is a constraint equation, xcex is an undetermined multiplier, and Qi is the force or torque applied to the ith coordinate.
In general, the prior systems and methods have been unable to provide realistic and like-like motions that are affordable. The complex processing requires very expensive and sophisticated equipment, and time-consuming.
There have been many attempts to alleviate the aforementioned problems and create a natural looking animation that is cost-effective and can be developed quickly and with increased creativity. The invention of PCT/US97/05513 ""513 is an animation system involving online computational methods for coordinating limb movements of articulated characters by solving associated forward and inverse kinematics problems in real time subject to multiple goals and constraints. The ""513 invention determines how the joints of an interactive character should rotate as a function of time to produce goal-directed limb movements.
The ""513 invention uses an on-line successive approximation method for solving inverse and forward kinematics problems concurrently, taking into account multiple goals and constraints. As a result, the ""513 invention is capable of producing fully interactive goal-directed behaviors in real-time for three-dimensional articulated characters with multiple limbs through the simultaneous satisfaction of position, alignment, posture, balance, obstacle avoidance, and joint limitation constraints.
In order to perform such animation, the ""513 invention performs separate calculations at each joint using goal-based motion primitives by a process which produces a set of joint angle movements that attempt to satisfy a higher-level goal, such as a positioning, alignment, posture, balance, obstacle avoidance, or joint limitation constraint. A geometric successive approximation process simplifies the problem of limb coordination.
One object of the present invention is an animation system that utilizes custom blending of motion sequences in real-time. The blending uses motion sequences from comparatives of a motion capture libraries as a starting point and customizes the sequences for the desired effect.
A further object is a system that uses inverse kinematics for linear computations of rotation (degrees) of all limbs. Forward dynamics are used to calculate a blend per limb that considers the temporal acceleration/velocity at any given frame of the source and any given frame of the destination. The blending results in a dynamic velocity based blending.
An object of the invention is a customizable animation system that provides network access to a database of recorded motion sequences and a means for customizing motion sequences over the network. Network in this context is defined any Internet, Intranet, and wireless interconnection between computer systems. An additional object is to provide access anywhere and anytime to the database and the tools required to customize motion sequences in real time.
A further object of the invention is a customizable animation system that permits a user to pay only for the sequence generated. Payment per frame or per motion sequence provides an inexpensive means of authoring new motion sequences at a fraction of the cost of other alternatives.
The concept of the blending is the capacity to predict the correct bio-mechanical behavior of the human body, as long as it is target driven. If for example you are sitting on a chair, and you decide to get up, it is possible to predict the vector of your perceived trajectory. There are an infinite number of ways to up from the sitting position, however, all the motion variances are created to facilitate the optimal force vector that gets your body to an upright standing position.
The process of going from a seated position to a standing position can be broken down as follows according to the mental processes: 1) Observing obstacles (floor space, nearby objects) and other objects to evaluate distance and assertive elements such as side arms. 2) Comparing an internally generated motion model of the perceived trajectory of getting up, relying on learned motion macro that is learned at an early age by trial and error. 3) Building an anticipatory force vector and begin activating muscle clusters leading to the buildup of perceived vector of optimum for the required standing motion and 4) Evaluating the next macro (eg; what to do after standing) before committing to the perceived vector. Committing to the xe2x80x98standing up macroxe2x80x99 and continuously evaluating the performance of the body.
The underlying premise of the behavioral blender of the present invention is that every human macro can be computed. The two main modules of the present invention are a motion capture library (as compared to the motion memories and xe2x80x98learnedxe2x80x99 macros) and two sets of rules. The two rule sets are internal and external. Internal rules refers to the kinematic skeletal bio-mechanical conditions of the bones, muscles, etc . . . The external rule set refers to the environmental conditions such as gravity, objects, and other dynamic forces.
The blender functions by first implementing the internal basic environment that allows blending between two or more sets of motion capture data, subject to the correct bio-mechanical behavior (eg: reaching the fulfillment of a set of conditions relating to marker placement based on skeleton segment length and other variables.
Next, a predictive algorithm is implemented to handle xe2x80x98perceivedxe2x80x99 optimal motion vector. (When a user is choosing the source and the destination of the blended sequence, the source""s velocities and accelerations are extrapolated, the reverse happens with the destination""s acceleration and velocities, The predictive algorithm offers the user the xe2x80x9cmost naturalxe2x80x9d blend interval in time. Finally, the environmental rule sets are implemented, facilitating realtime simulations of the velocity/accelerations variances in response to a changing conditions in the environment. Those Simulations make use of Lagrange equations in combinations with the (above discussed) velocity base blends.
The present invention has several components, including a motion database, blender software, custom user interface. A further description of the blending algorithm and template matching as well as a detailed description of motion capture in general is incorporated by reference from the published application PCT/US99/21246 to the same applicant.
The motion capture database comprises an entire full body setup. All the compensatory motions of the body are captured while isolating one body part for the eventual blends. The database contains many general or standard motions such as walking, running, sitting, standing, falling, recoveries, gestures, etc . . .
The blender software computes the blends between any matching elements of the motion databases in a manner that complies with the over-riding bio-mechanical laws. Blends that result in motion that are not natural or are otherwise impossible for the human body to execute are extrapolated to fit a correct behavior.
Another embodiment of the present invention is through the use of blender software using genetic algorithms. Instead of looking at blending motion capture databases according to bio-mechanical laws, parsing those motion sequences through a genetic algorithm and presenting multiple possibilities of variations of thew original motions. The operator can then choose the preferred xe2x80x98mutationxe2x80x99 and parse that sequence through same algorithm to create a new set of variations. Each new selection introduces a new set of mutations for operator selection.
The custom user interface is a GUI interface that allows full access of all subsets and blending methods, or as otherwise prescribed by agreement or license.
The present invention is an Internet product that is downloadable for implementation. The motion capture libraries are also accessible via the Internet. A user can obtain any range of motion capture data and then use the software of the present invention to custom tailor the animation for the desired effect.
For example, if the libraries have a user sequence walking at one pace and another user sequence walking at a different pace, the blender can create any subset between those two data sequences. It is not as simple as an interpolation between data setsxe2x80x94the over-riding laws of bio-mechanics must be accounted for in creating the custom set. As a person walks, there are many forces at work and moving parts. Changing one element of the mix affects all the other elements.
The motion capture data sequences are obtained using sensors attached to various parts of the body. The movements can be plotted a three dimensional space. The blending between two sequences is not a simple interpolation between the plots because the actual trajectory and bio-mechanical laws dictate the outcome.
There are current systems of animation that comply with the bio-mechanical laws, but the resulting motions do not appear natural. The introduction and blending of the motion capture data into the mix creates life-like and natural movements.
Custom motion capture also exists, but involves extraordinary time and expense to create a sequence for only a particular movement. Clip art works for generic images or animation, such as famous landmarks, but does not permit custom tailoring.
Cost advantages of a clip art library that allows you to customize any work create an original work. For example, a walking sequence can be extracted from the library. The user can then custom-tailor the swinging arm movement. The user may then introduce a swinging head motionxe2x80x94which in turn affects the swinging arm movement and the walk. Every tweak by the user creates a new and original work.
Another method of customizing animation is through genetic algorithms. It is a separate way to look at blending where mutations are created depending on the input selections. Mutations of the motion capture data get generated from the behavioral data and a set of different mutations are created for each behavioral alteration. Applying genetic algorithms to variances in motion capture data is another object of the present invention and will yield very different results from the blending.
The blender algorithm uses forward and inverse kinematic equations and Lagrange equations to calculate and determine the joint torque and joint forces at each degree of animation. Although the human body is susceptible to over 200 degrees of freedom, in reality, the number is closer to 40 for realistic applications.
The present invention also describes the use of genetic algorithms the necessary equations and calculations that are modeled in the algorithms for the present invention.
In contrast to the PCT application WO9740471 (""471) application, the present invention accomplishes that which is described in the ""471 application as xe2x80x9cpractically impossiblexe2x80x9d due to the computationally intensive matrix inverse calculations. The present invention allows inverse kinematic methods to interactively animate any realistic human or animal-like character in real time using both forward kinematics and inverse kinematics.
In distinction, the ""471 invention does not use matrix inverse calculations and has computational requirements that scale linearly with the number of joints. Scaling linearly with the number of joints enables the ""471 invention to interactively animate moderately complex articulated characters in real time using commonly available personal computers.
Whereas traditional inverse kinematic methods converge to a solution (usually defined as a minimum error condition) before the results of the computation can be used to animate the character. In contrast, the ""471 invention is an iterative computational procedure that always produces somewhat meaningful joint motion (with no singularity conditions) after each iteration of the computation. It is not necessary to wait for the ""471 process to converge to a minimum error condition. The results of the calculation can be used xe2x80x98anytimexe2x80x9d, i.e., within the constraints of a real time animation system. The ""471 methods tolerate more tracking error, i.e., greater position, velocity or acceleration error, than matrix inverse methods, but in return provide more natural motion, and faster calculations. The present invention operates in real-time with less tracking error.
Some of the ""471 animation techniques are interactive and others are not. The ""471 techniques of keyframing and motion capture produce fixed, i.e., scripted and non-interactive motion data. In distinction, the present invention encompasses more interactive elements. And, the present invention integrates interactive motion data with interactive control, which is not disclosed or mentioned in the ""471
The demand for motion capture has been growing exponentially. Unfortunately it has been too expensive and too laborious for most applications. The browser of the present invention allows any person to be an animator and blend motion sequences on the high speed servers of the Internet based product. When the sequence is completed, the user simply pays for the 3D data and downloads the data to the users workstation. The software has been benchmarked and works at 15fps on a 56k modem.
A significant advantage of the present invention is the implementation and method of incorporating the system. The present invention has a database of motion capture data that is accessible on-line over an intranet or the Internet. The blending algorithms, either inverse kinematics or genetic algorithms, can be resident on the user""s computer system either from purchasing the software or downloaded over the Internet. A further feature of the present invention is to allow on-line and real-time manipulation of the motion capture library on-line. An individual license can be purchased for a particular animation and a new work created by a user on-line, making this a cost-effective and speedy transaction.
Accessing a motion capture library, the present invention allows the user to choose and xe2x80x9cblendxe2x80x9d motion sequences including individual body parts, until the correct effect is created. The end result is infinitely more flexible than xe2x80x9ccannedxe2x80x9d motion capture sequences, and extremely cost-effective as compared to studio production.
The present invention brings big production motion capture to projects lacking big production budgets, and makes the motion capture animation project faster, easier, and less costly to complete. This system provides a breakthrough in price/performance for 3D animators. Lifelike sequences are built in a fraction of the normal time, at approximately one-tenth the cost of traditional studio production
The present invention allows for each human body part motion to be altered independently. The user manipulates a huge library of captured data to produce unique sequences. The blending tools also marries individual sequences to produce smooth, lifelike movements, employing inverse kinematics.
As a networked system, the present invention is accessed through the Internet and is available anywhere and anytime.
In addition, the data sets created by the present invention can be directly exported into a wide range of professional 3D packages including products from SOFTIMAGE(trademark), Discreet(trademark), AliasIWavefront(trademark) and Side-Effects(trademark).
One of the unique features of the present invention is that is allows on-the-fly data sets viewed through the xe2x80x9cblenderxe2x80x9d window. Selected sequences and movements are reviewed in real-time before you commit to purchase and download.
Furthermore, the algorithms continually enforces bio-mechanical rules so that sequences maintain the ultimate in natural physical motion. For even more creativity of movements for non-human characters or to deliberately portray unreal human motion, the rules can be bypassed.
The motion capture data bank is a growing depository of motion capture data. As more data is captured the database will continue to expand.
Another object of the invention is a simple-to-understand interface, allowing the user to quickly select and manipulate motion sequences, then fine tune each body part using slider bars. The graphical user interface permits an untrained user to begin from basic building blocks and customize these animations according to individual preferences.
There are only a few simple steps for using the network based system. First, the user must access the blender download site and download the client browser. The user then selects one or more motion sequences from the data library. Using slider bars, the user modifies and fine tunes individual body parts and manipulates the motion sequences until the desired result is achieved. Once the user is satisfied with the sequence, the user selects a format for exporting the data. The user then selects from the various payment methods and chooses the format for downloading the data. Finally, the user imports the data to the individual animation tool or environment.
The foregoing description of the preferred embodiment of the invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.